-
Notifications
You must be signed in to change notification settings - Fork 44
Allow limited inbound ICMP to Nexus, add ICMP type/code filters to firewall rules #8194
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
5594fc0
to
38bd2ff
Compare
That seems to be be the primary reason for the entire test suite having a Very Bad Time.
Thanks, CRDB.
I've setup these bits on On the actual MTU dscovery front, it looks as though both this PR and If from the web console I perform a 'Reload (Override Cache)' we can kick off large enough TCP transfers to run into LSO. When we trace this through: DublinBRM23230018 # dtrace -n 'xde_mc_tx:entry{ this->m = (mblk_t *)arg1; this->mss = this->m->b_datap->db_struioun.cksum.pad } xde_mc_tx:entry/this->m->b_datap->db_struioun.cksum.flags && this-
>mss != 0xBADD && this->mss/{printf("packet with MSS %d", this->m->b_datap->db_struioun.cksum.pad);}'
dtrace: description 'xde_mc_tx:entry' matched 2 probes
CPU ID FUNCTION:NAME
111 5301 xde_mc_tx:entry packet with MSS 1368
69 5301 xde_mc_tx:entry packet with MSS 1368
69 5301 xde_mc_tx:entry packet with MSS 1368
69 5301 xde_mc_tx:entry packet with MSS 1368
69 5301 xde_mc_tx:entry packet with MSS 1368
69 5301 xde_mc_tx:entry packet with MSS 1368
69 5301 xde_mc_tx:entry packet with MSS 1368
69 5301 xde_mc_tx:entry packet with MSS 1368
69 5301 xde_mc_tx:entry packet with MSS 1368
69 5301 xde_mc_tx:entry packet with MSS 1368
69 5301 xde_mc_tx:entry packet with MSS 1368
69 5301 xde_mc_tx:entry packet with MSS 1368
69 5301 xde_mc_tx:entry packet with MSS 1368
69 5301 xde_mc_tx:entry packet with MSS 1368
69 5301 xde_mc_tx:entry packet with MSS 1368
1 5301 xde_mc_tx:entry packet with MSS 1368
69 5301 xde_mc_tx:entry packet with MSS 1368
69 5301 xde_mc_tx:entry packet with MSS 1368
^C
BRM23230018 # dtrace -n 't4_eth_tx:entry{ this->m = (mblk_t *)arg1; this->mss = this->m->b_datap->db_struioun.cksum.pad } t4_eth_tx:entry/this->m->b_datap->db_struioun.cksum.flags && this-
>mss != 0xBADD && this->mss/{printf("packet with MSS %d", this->m->b_datap->db_struioun.cksum.pad);}'
dtrace: description 't4_eth_tx:entry' matched 2 probes
CPU ID FUNCTION:NAME
9 6360 t4_eth_tx:entry packet with MSS 8928
34 6360 t4_eth_tx:entry packet with MSS 8928
117 6360 t4_eth_tx:entry packet with MSS 8928
120 6360 t4_eth_tx:entry packet with MSS 8928
110 6360 t4_eth_tx:entry packet with MSS 8928
120 6360 t4_eth_tx:entry packet with MSS 8928
86 6360 t4_eth_tx:entry packet with MSS 8928
86 6360 t4_eth_tx:entry packet with MSS 8928
77 6360 t4_eth_tx:entry packet with MSS 8928
78 6360 t4_eth_tx:entry packet with MSS 8928
87 6360 t4_eth_tx:entry packet with MSS 1368
8 6360 t4_eth_tx:entry packet with MSS 8928
69 6360 t4_eth_tx:entry packet with MSS 1368
69 6360 t4_eth_tx:entry packet with MSS 1368
69 6360 t4_eth_tx:entry packet with MSS 1368
78 6360 t4_eth_tx:entry packet with MSS 8928
69 6360 t4_eth_tx:entry packet with MSS 1368
69 6360 t4_eth_tx:entry packet with MSS 1368
69 6360 t4_eth_tx:entry packet with MSS 1368
69 6360 t4_eth_tx:entry packet with MSS 1368
43 6360 t4_eth_tx:entry packet with MSS 1368
69 6360 t4_eth_tx:entry packet with MSS 1368
69 6360 t4_eth_tx:entry packet with MSS 1368
69 6360 t4_eth_tx:entry packet with MSS 1368
69 6360 t4_eth_tx:entry packet with MSS 1368
69 6360 t4_eth_tx:entry packet with MSS 1368
43 6360 t4_eth_tx:entry packet with MSS 1368
43 6360 t4_eth_tx:entry packet with MSS 1368
69 6360 t4_eth_tx:entry packet with MSS 1368
69 6360 t4_eth_tx:entry packet with MSS 1368
69 6360 t4_eth_tx:entry packet with MSS 1368
78 6360 t4_eth_tx:entry packet with MSS 8928
69 6360 t4_eth_tx:entry packet with MSS 1368
69 6360 t4_eth_tx:entry packet with MSS 1368
43 6360 t4_eth_tx:entry packet with MSS 1368
43 6360 t4_eth_tx:entry packet with MSS 1368
^C The latter query is also catching underlay traffic, hence the 8928B entries. An MSS of 1368 matches what I'd maybe expect for a reduced MTU of 1402 bytes (resp. 1448 for a 1500B MTU). Do we have any hits on the new rules? BRM23230018 # opteadm dump-layer -p opte0 firewall
Port opte0 - Layer firewall
======================================================================
Inbound Flows
----------------------------------------------------------------------
PROTO SRC IP SPORT DST IP DPORT HITS ACTION
TCP 172.20.17.94 50056 172.30.2.6 443 0 no-op
TCP 172.20.17.94 50062 172.30.2.6 443 0 no-op
Outbound Flows
----------------------------------------------------------------------
PROTO SRC IP SPORT DST IP DPORT HITS ACTION
TCP 172.30.2.6 443 172.20.17.94 50056 1 no-op
TCP 172.30.2.6 443 172.20.17.94 50062 1 no-op
Inbound Rules
----------------------------------------------------------------------
ID PRI HITS PREDICATES ACTION
126 65534 2 inner.ip.proto=TCP "Stateful Allow"
inner.ulp.dst=80,=443
125 65534 0 inner.icmp.type=time exceeded "Stateful Allow"
124 65534 0 inner.icmp.type=destination unreachable "Stateful Allow"
inner.icmp.code=4
DEF -- 427 -- "deny"
Outbound Rules
----------------------------------------------------------------------
ID PRI HITS PREDICATES ACTION
DEF -- 2 -- "stateful allow" Apparently not, although it's hard to tell given that the reconcile task is completely reinstating the rules periodically so we're losing those counters. Note that all the DogfoodBRM44220011 # dtrace -n 'xde_mc_tx:entry{ this->m = (mblk_t *)arg1; this->mss = this->m->b_datap->db_struioun.cksum.pad } xde_mc_tx:entry/this->m->b_datap->db_struioun.cksum.flags && this-
>mss != 0xBADD && this->mss/{printf("packet with MSS %d", this->m->b_datap->db_struioun.cksum.pad);}'
dtrace: description 'xde_mc_tx:entry' matched 2 probes
CPU ID FUNCTION:NAME
41 4868 xde_mc_tx:entry packet with MSS 1448
9 4868 xde_mc_tx:entry packet with MSS 1368
68 4868 xde_mc_tx:entry packet with MSS 1368
68 4868 xde_mc_tx:entry packet with MSS 1368
68 4868 xde_mc_tx:entry packet with MSS 1368
68 4868 xde_mc_tx:entry packet with MSS 1368
68 4868 xde_mc_tx:entry packet with MSS 1368
68 4868 xde_mc_tx:entry packet with MSS 1368
68 4868 xde_mc_tx:entry packet with MSS 1368
68 4868 xde_mc_tx:entry packet with MSS 1368
127 4868 xde_mc_tx:entry packet with MSS 1368
68 4868 xde_mc_tx:entry packet with MSS 1368
68 4868 xde_mc_tx:entry packet with MSS 1368
68 4868 xde_mc_tx:entry packet with MSS 1368
68 4868 xde_mc_tx:entry packet with MSS 1368
68 4868 xde_mc_tx:entry packet with MSS 1368
68 4868 xde_mc_tx:entry packet with MSS 1368
68 4868 xde_mc_tx:entry packet with MSS 1368
68 4868 xde_mc_tx:entry packet with MSS 1368
68 4868 xde_mc_tx:entry packet with MSS 1368
68 4868 xde_mc_tx:entry packet with MSS 1368
83 4868 xde_mc_tx:entry packet with MSS 1368
83 4868 xde_mc_tx:entry packet with MSS 1368
83 4868 xde_mc_tx:entry packet with MSS 1368
83 4868 xde_mc_tx:entry packet with MSS 1368
^C
BRM44220011 # dtrace -n 't4_eth_tx:entry{ this->m = (mblk_t *)arg1; this->mss = this->m->b_datap->db_struioun.cksum.pad } t4_eth_tx:entry/this->m->b_datap->db_struioun.cksum.flags && this->mss != 0xBADD && this->mss/{printf("packet with MSS %d", this->m->b_datap->db_struioun.cksum.pad);}'
dtrace: description 't4_eth_tx:entry' matched 2 probes
CPU ID FUNCTION:NAME
18 5927 t4_eth_tx:entry packet with MSS 8928
52 5927 t4_eth_tx:entry packet with MSS 8928
52 5927 t4_eth_tx:entry packet with MSS 1448
113 5927 t4_eth_tx:entry packet with MSS 1368
68 5927 t4_eth_tx:entry packet with MSS 1368
68 5927 t4_eth_tx:entry packet with MSS 1368
68 5927 t4_eth_tx:entry packet with MSS 1368
76 5927 t4_eth_tx:entry packet with MSS 8928
68 5927 t4_eth_tx:entry packet with MSS 1368
68 5927 t4_eth_tx:entry packet with MSS 1368
68 5927 t4_eth_tx:entry packet with MSS 1368
68 5927 t4_eth_tx:entry packet with MSS 1368
68 5927 t4_eth_tx:entry packet with MSS 1368
68 5927 t4_eth_tx:entry packet with MSS 1368
68 5927 t4_eth_tx:entry packet with MSS 1368
44 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
44 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
44 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
44 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
44 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
44 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
44 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
44 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
44 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
44 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
44 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
44 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
44 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
44 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1448
44 5927 t4_eth_tx:entry packet with MSS 1448
68 5927 t4_eth_tx:entry packet with MSS 1368
68 5927 t4_eth_tx:entry packet with MSS 1368
68 5927 t4_eth_tx:entry packet with MSS 1368
68 5927 t4_eth_tx:entry packet with MSS 1368
18 5927 t4_eth_tx:entry packet with MSS 1368
18 5927 t4_eth_tx:entry packet with MSS 1368
68 5927 t4_eth_tx:entry packet with MSS 1368
68 5927 t4_eth_tx:entry packet with MSS 1368
68 5927 t4_eth_tx:entry packet with MSS 1368
68 5927 t4_eth_tx:entry packet with MSS 1368
120 5927 t4_eth_tx:entry packet with MSS 1368
68 5927 t4_eth_tx:entry packet with MSS 1368
68 5927 t4_eth_tx:entry packet with MSS 1368
68 5927 t4_eth_tx:entry packet with MSS 1368
68 5927 t4_eth_tx:entry packet with MSS 1368
68 5927 t4_eth_tx:entry packet with MSS 1368
^C
76 5927 t4_eth_tx:entry packet with MSS 8928
95 5927 t4_eth_tx:entry packet with MSS 1368
78 5927 t4_eth_tx:entry packet with MSS 1368
43 5927 t4_eth_tx:entry packet with MSS 1368
68 5927 t4_eth_tx:entry packet with MSS 1368
43 5927 t4_eth_tx:entry packet with MSS 1368
43 5927 t4_eth_tx:entry packet with MSS 1368
60 5927 t4_eth_tx:entry packet with MSS 1368
43 5927 t4_eth_tx:entry packet with MSS 1368
60 5927 t4_eth_tx:entry packet with MSS 1368
60 5927 t4_eth_tx:entry packet with MSS 1368
1 5927 t4_eth_tx:entry packet with MSS 1368
113 5927 t4_eth_tx:entry packet with MSS 1368
1 5927 t4_eth_tx:entry packet with MSS 1368
69 5927 t4_eth_tx:entry packet with MSS 1368
86 5927 t4_eth_tx:entry packet with MSS 1368
52 5927 t4_eth_tx:entry packet with MSS 1368
68 5927 t4_eth_tx:entry packet with MSS 1368
42 5927 t4_eth_tx:entry packet with MSS 1368
42 5927 t4_eth_tx:entry packet with MSS 1368
42 5927 t4_eth_tx:entry packet with MSS 1368
103 5927 t4_eth_tx:entry packet with MSS 1368
19 5927 t4_eth_tx:entry packet with MSS 1368
19 5927 t4_eth_tx:entry packet with MSS 1368
68 5927 t4_eth_tx:entry packet with MSS 1368
68 5927 t4_eth_tx:entry packet with MSS 1368 There's a mix in there, but I assume we have something on the lab-side hitting the API periodically, since I see those 1448 entries passively. But dogfood appears to be discovering and using the reduced MTU just fine. Fixing oxidecomputer/opte#748 may have been sufficient to get this working, and that PR has been pulled into |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this is good for review, I've pointed out some open design points I'm unsure on.
@@ -136,7 +136,7 @@ z_swadm () { | |||
|
|||
# only set this if you want to override the version of opte/xde installed by the | |||
# install_opte.sh script | |||
OPTE_COMMIT="" | |||
OPTE_COMMIT="792ec8a3816ba8c7c4268f65cd19dbd946a9027d" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Noting that this needs to be removed as part of merging.
// Port assignments are incompatible with non | ||
// TCP/UDP protocols. | ||
if matches!( | ||
proto, | ||
ProtoFilter::Tcp | ProtoFilter::Udp | ||
) { | ||
filters.set_ports(ports.clone()); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm surprised we haven't run into this one yet -- if someone were to install a rule on [TCP, ICMP] x [port 80, port 443]
this would previously be installed as the two rules:
- proto = TCP && (port = 80 || port = 443), [valid]
- proto = ICMP && (port = 80 || port = 443). [invalid]
In OPTE, a port match will always return false unless it is applied to TCP/UDP traffic.
targets: vec![VpcFirewallRuleTarget::Subnet( | ||
super::vpc_subnet::NEXUS_VPC_SUBNET.name().clone(), | ||
)], |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we want to include any more services here?
protocols: Some(vec![ | ||
VpcFirewallRuleProtocol::Icmp(Some(VpcFirewallIcmpFilter { | ||
// Type 3 -- Destination Unreachable | ||
icmp_type: 3, | ||
// Code 4 -- Fragmentation needed | ||
code: Some(4.into()), | ||
})), | ||
VpcFirewallRuleProtocol::Icmp(Some(VpcFirewallIcmpFilter { | ||
// Type 11 -- Time Exceeded | ||
icmp_type: 11, | ||
code: None, | ||
})), | ||
]), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This gives us PTMUD (although it looks as though the TCP stack can figure it out in the absence of Fragmentation Needed messages?), and the ability to use traceroute
from Nexus/DNS. Is the latter needed? Are there any other ICMP families which would be crucial?
nexus/external-api/src/lib.rs
Outdated
async fn networking_inbound_icmp_view( | ||
rqctx: RequestContext<Self::Context>, | ||
) -> Result<HttpResponseOk<bool>, HttpError>; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This works on a bool
today to have a simple toggle. Would we rather have a Vec<VpcFirewallIcmpFilter>
(or similar) here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In general, I think we prefer to avoid JSON bodies that are just a single primitive type not wrapped in a response object as it's caused issues with client coe generation the past; see the final paragraph in this comment from @david-crespo:
A related but distinct issue is that we generally want API request and response bodies to be JSON objects rather than strings. We've run into issues with client generation when that has not been the case. A string is valid JSON, but I think this would be the only endpoint that returns a string. One reason to prefer objects is that objects can be extended by adding another key without changing the basic shape of the thing. If these unlikely to change their shape, consistency with other endpoints is probably a stronger argument.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I think { "enabled": boolean }
would be better. In addition to the above excellent points, it's a nice extra bit of unavoidable documentation about what the boolean represents.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That sounds good to me, it should be a simple enough change. I wasn't aware of the impact on clients here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should be done in 5c7caed.
String::from_sql(bytes)? | ||
.parse::<external::VpcFirewallRuleProtocol>()?, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i had a vague memory of there being some way to do this by turning the bytes into an &str
with str::from_utf8
or something, so that we don't make a whole String
just to parse it and throw it away...but i wasn't able to find prior art for that elsewhere in Nexus, so i dunno if it's worth the effort.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I had a quick look there and I think our options are limited here -- I couldn't find any other obvious examples in nexus itself, either.
Frustratingly, PgValue
does have the right method for us, but this is not part of any trait shared by Backend::RawValue
s. We also don't have any FromSql<Text, _>
for &[u8]
/*const [u8]
. The next best thing is, seemingly, deserializing to a *const str
. Kinda gross, and quoth the docs:
We have to return a raw pointer instead of a reference with a lifetime due to the structure of FromSql
Less than ideal. It would work great in a blanket impl for all FromStr
types (where we could justify the unsafe
), except I think the orphan rule would prevent us from trivially doing so. I suspect there's something clever we could do, but maybe upending everyone's FromSql<sql_types::Text, DB>
impls is a bridge too far in this PR? I kinda want to see if it can be done in a followup.
This PR adds more detail to firewall protocol filters. Rather than being specified purely as a string->enum mapping, each protocol filter is now a standard tagged enum like many other types in the Nexus API. Here, this allows us to express more control over ICMP traffic. For instance:
[{"type": "icmp"}]
),[{"type": "icmp", "value": { icmp_type: 0 }}, {"type": "icmp", "value": { icmp_type: 8 }}]
){"type": "icmp", "value": { icmp_type: 0, code: "0-4" }}
).Building on this, Nexus now has a firewall entry to permit the receipt of e.g., Destination Unreachable messages. I've added a new system endpoint to control whether this is enabled or disabled.
As part of this, I've converted the CRDB representation from a fixed
ENUM[]
to aSTRING(32)[]
. There are a couple of reasons for this:Should close #7998.